You have several options to travel from Malaysia to Cambodia and it depends on your time and budget.
The fastest option:
Fly to Phnom Penh from Kuala Lumpur and there are several flight companies who offer daily flights. The one we personally use is Air Asia but be aware that they have a 15 KG baggage weight limit. At a cost of about USD20, you can get an additional 15 KG if you check in a bag as a SPORTS BAG. Check www.airasia.com where you can book online.
The slower options:
By bus or train