Bifidobacteria are early colonizers of the human neonatal gut and provide multiple health benefits to the infant, including inhibiting the growth of enteropathogens and modulating the immune system. Certain Bifidobacterium species prevail in the gut of breastfed infants due to the ability of these microorganisms to selectively forage glycans present in human milk, specifically human milk oligosaccharides (HMOs) and N-linked glycans. Therefore, these carbohydrates serve as promising prebiotic dietary supplements to stimulate the growth of bifidobacteria in the guts of children suffering from impaired gut microbiota development. However, the rational formulation of milk glycan-based prebiotics requires a detailed understanding of how bifidobacteria metabolize these carbohydrates. Accumulating biochemical and genomic data suggest that HMO and N-glycan assimilation abilities vary remarkably within the Bifidobacterium genus, both at the species and strain levels. This review focuses on the delineation and genome-based comparative analysis of differences in respective biochemical pathways, transport systems, and associated transcriptional regulatory networks, providing a foundation for genomics-based projection of milk glycan utilization capabilities across a rapidly growing number of sequenced bifidobacterial genomes and metagenomic datasets. This analysis also highlights remaining knowledge gaps and suggests directions for future studies to optimize the formulation of milk-glycan-based prebiotics that target bifidobacteria.