Q] Why didn't SEGA just port the arcade game?
A] It isn't clear -perhaps the arcade game didn't do that well, so they wanted to change it for the Genesis. This isn't the only time SEGA have done something like this, as the arcade and Genesis versions of ESWAT are very different from each other.
Q] Where does Yamato come from?
A] The American manual expands on the backstory a bit, mentioning things which aren't talked about in the game itself. It states that After Joe defeated the Neo Zeed, he briefly went to New York to see his student, Kato, before returning to Japan. As Joe went there, Kato heard that a school was attacked and hostages had been taken. Kato attempts to rescue them but is severely wounded. Joe finds out about this and immediately returns to New York. And to quote the manual exactly: "But before he could get to the hospital, Kato passed away, leaving nothing but memories and his faithful dog, Yamato. Overcome with grief and rage, Joe Musashi promised Kato's mother that he would avenge her son's death."
The manual then mentions that Kato was killed by a ninja warrior of the Union Lizard. But to answer the question, Yamato belonged to Kato, who was Joe's student. Absolutely none of this is mentioned in the game itself. I don't know if Kato is mentioned in the Japanese version's manual, either.
Q] Any prototype screenshots?
A] Here are a few:
This screenshot shows a possible early title screen for Shadow Dancer, featuring Hayate/Joe Musashi.
These two images both have different graphics for Yamato's gauge. But more interestingly is the fact that Joe seems to be using some kind of lightning bolt ninjitsu in the first screen! This magic attack is not present in the final version. Thanks to the staff of this site for these shots.
These two screenshots are on the back of the box for the Brazilian version of Shadow Dancer. There admittedly isn't anything that different about them other than the sprite for Yamato's gauge, which looks like a shuriken. Thanks to SEGA Retro for the scans.
The American version's manual refers to round 5 -which is called "Union Lizard" in the game -as "The Reptile Sauros". Was this the original name for this round, or just a mistake? The manual gets all of the other round names correct.
Q] Does changing the level option effect anything other than the difficulty?
A] The endings of the game are slightly different depending on what level setting it was beaten on. Level 1 is this, level 2 is this and finally level 3 is this. Also, the "Secret of Shinobi" logo shown before the intro will become blue on levels 2 and 3, rather than the red used for level 1.