It means “What the hell is this?”
“这是什么鬼” can be regarded as a short form of “这是什么鬼东西”. The “鬼”, literally means ghost or monster, is used to express one’s surprise on something seemingly unreasonable or embarrassing. “这是什么（东西）” means “What’s this (thing)”, so “这是什么鬼” means “What the hell is this”