I usually pick a theme or a few topics to base it around and then find characters I like that could fit in. (Although sometimes I do this the other way around and end up with the characters first, then find things they have in common).
Once I have a basic idea of what I want (say a fantasy world with dragons or something), I start coming up with locations and concepts for the landscape. Since I mainly do scenery, this is where it starts getting more artsy for me, but the majority of it is still written descriptions.
So, we have a fantasy world with dragons. Well, maybe those dragons like mountains, so there's going to have to be a mountain range somewhere. To get the actual world map drawn out, I usually drop a handful of something on a paper and loosely trace around it (beads, rice, dice, whatever) to get a shape for the land areas. If using dice, you can assign a number to a certain feature if you want to and the same goes for bead colors. Or you can just put things randomly like I do most of the time :')
So, now you have land, characters, an idea, and maybe some mountains somewhere on that land. From there, I usually start planning where I want the other places I came up with during the concept-ish part to go and how they would relate to each other. I also try asking some questions about how everything interacts (Would the dragons be near a kingdom? Would the humans try fighting them? What effects would this have? Where else could dragons live? Are the dragons important to the humans at all?) to figure out placement as well as who gets along with who and what groups should, probably be separated. This helps with boundaries.
Usually I stop at the basics (lore comes after the boundaries, since it kind of builds on everything else), but with some of the more detailed worlds I will continue into things like political structures, detailed history (dates, specific events, etc), religion, and beliefs / traditions/ behaviors.
Overall, I can do things in pretty much any order, but this is my usual way. Although I do skip around on some of the more complex ones and come up with storylines first, like some of the others have mentioned.
It's interesting to see how others do this, too :O