It is a known fact that excessive estrogen production in men can cause development of breasts, known as gynecomastia, nickname moobs (male boobs).
Researchers have been looking at the possible causes of excessive estrogen production “in  vitro” (an artificial environment outside the human body) and “in vivo” within the human body. And the result is inconclusive as with many studies. Why will the very same conditions effect one man and not another? No one knows. However it is interesting to be aware of what researchers are looking at and then you can make a decision for yourself.
Hormone treatment in animals has long been believed to cause breast development in men, and therefore have been outlawed in most countries for years, (Australia’s beef however is an exception). The incidence of “moobs” is still on the increase causing researcher’s to continue their quest to find the cause or causes.   �
In the last couple of years, lavender oil and tea tree oil have come under scrutiny. A report in WebMD whose source is The New England Journal of Medicine states:
(WebMD)  Repeated use of products containing lavender oil or tea tree oil may spur breast growth in prepubertal boys, experts report in The New England Journal of Medicine.
“This report raises an issue of concern, since lavender and tea tree oil are sold over the counter in their ‘pure’ form and are present in an increasing number of commercial products, including shampoos, hair gels, soaps, and body lotions,” write researchers Derek Henley, Ph.D., and others. Henley works at the National Institute of Environmental Health Sciences (NIEHS), which is part of the National Institutes of Health (NIH).

“We want to encourage doctors who may be seeing patients with gynecomastia to ask their patients about the products they are using,” says Henley’s colleague, Kenneth Korach, PhD, in a NIEHS news release. “Patients with prepubertal gynecomastia may want to consider reducing the use of products that contain these oils,” says Korach.

Henley and colleagues investigated reports of three healthy boys who hadn’t started puberty and who developed breasts, a condition called gynecomastia, after using products containing lavender or tea tree oils.

It’s rare for prepubertal boys to develop breasts, the researchers note.

One of the boys was 4 years old. His mother had applied “healing balm” containing lavender oil to his skin shortly before the boy’s breasts developed and grew to about an inch in diameter.

Another boy was 10 and had been using a hair-styling gel daily that contained lavender and tea tree oils.

The third boy — nearly 8 — had been using lavender-scented soap and skin lotions. His fraternal twin had used the same products but didn’t develop breasts.

The boys’ breasts receded within months after they stopped using the products.

Henley’s team conducted lab tests on human breast cells exposed to lavender and tea tree oils. The tests showed that lavender and tea tree oils may boost estrogen (a sex hormone that promotes female characteristics and is linked to breast development) and hamper androgens (sex hormones that promote male characteristics and inhibit breast growth).

The researchers call for more observational studies to track prepubertal gynecomastia in boys using such products.

Henley and colleagues don’t mention specific products in their report, and they’re not ruling out the possibility that the boys’ breast growth may have stemmed from other causes.

It’s often hard to pinpoint the exact cause of gynecomastia, the researchers note.

However, “We conclude that repeated topical exposure to lavender and tea tree oils probably caused prepubertal gynecomastia in these boys,” write Henley and colleagues.
“We do not anticipate any long-term effects on hormone levels” in the boys, Henley says in an NIEHS news release.
SOURCES: Henley, D. The New England Journal of Medicine, Feb. 1, 2007; Vol. 356: pp. 479-485. News release, National Institute of Environmental Health Sciences, National Institutes of Health.

And then Wikipedia has this to say about the above report:
CONTRAINDICATIONS
Lavender oil has recently been implicated in gynecomastia, the abnormal development of breasts in young boys. Denver endocrinologist Clifford Bloch hypothesized the link after three boys presented with enlarged breasts. Subsequently, Derek Henley and Kenneth Korach of the National Institute of Environmental Health Sciences in Research Triangle Park, N.C., discovered in lavender and tea tree oil the presence of compounds which both suppress male hormones and mimic female hormones.
Because sex hormone levels normally are low prior to puberty, young boys and girls particularly are sensitive to estrogenic and androgenic compounds. The discovery of the gynecomastia link in boys has led some researchers to suspect lavender and tea tree oils, which are present in various personal care products including shampoos and lotions, may also contribute to the increased incidence of early breast development in girls.
Discontinuation of use of these products resulted in rapid reversal of gynecomastia in Bloch’s young patients.[4][5]
However, the conclusion that the gynecomastia actually was caused by the essential oils in the products used by the three boys are currently being disputed by the Natural Artisan Perfumers Guild and Cropwatch on the claimed basis of insufficient evidence.

As you can see the jury is still out on the causes of gynecomastia for the individual man. The best advice is to consult with your doctor and go over your personal medical history in detail to determine your best course of action.