Does the buckle matter when determining belt size?

Yes!  With any belt, the buckle is part of the belt size.  Consider the following picture.  Three straps are shown, each the same size. Changing the buckle to a buckle of different length changes the belt size by that change in length.

belt buckle sizing
Effect of buckle size on belt size.