The musician Patti Smith was desperate to work with: “We fought a lot”