We intuitively grasp a kind of unity of each human person, but have no special, privileged mode of knowledge of persons as individuals. Common sense tends to be rather dogmatic, and glosses over many distinctions in such matters. Plato compared the soul to a city, a sort of community of thoughts and desires — a kind of unity to be sure, but a relatively weak one. In Kantian terms, human persons seem to be distinguished from everything else by somehow being the nexus of combination of otherwise very distinct empirical and transcendental domains.
Considerations of change over time further complicate the picture, but may also provide a kind of guiding thread. A factual “me” is mainly a retrospective construction. A normative “I” on the other hand has both retrospective and prospective aspects. Brandom’s and Pippin’s readings of Hegel emphasize that we should think of agency and acts as always comprising both a partially constituted, retrospectively constructed past and a yet-to-be-determined future. Ricoeur has developed a temporally extended, retrospective and prospective notion of self as an ethical aim or promise rather than an existing actuality. Such an aim or promise, it seems to me, can have a much stronger unity than we could legitimately claim as an existing actuality.
Rather than conflating the empirical and transcendental, as in the Latin medieval notion of an “intellectual soul” — or inflating a notion of empirical self to fill the whole space of subjectivity, in the common modern way — we can tie the unification of empirical and transcendental elements to that prospective aim or promise, without asserting it in the present. (See also Empirical-Transcendental Doublet?; Two Kinds of Character; Narrated Time; Hegel’s Ethical Innovation; Hegel on Willing.)