From hexdump

Anything where the hex bytes are pairs of hex digits on a line. Before reading a line it removes a trailing ASCII column of the form |....| and a leading offset column matching 6 to 16 hex digits with an optional 0x prefix and an optional colon. What is left is scanned for two-digit hex pairs. Plain hex text with no columns at all works too, so a blob copied out of a debugger converts directly.

Because no line held a pair of hex digits after the columns were stripped. The full message is that no hex byte pairs were found and a hexdump-style text file was expected. Usual causes: you dropped the binary itself rather than a text dump, the file is base64 rather than hex, or the dump uses a format such as decimal or octal bytes.

Most likely stray hex-looking text was read as data. The ASCII column is only stripped when it is wrapped in pipes, so a dump whose text column has no pipes can contribute extra bytes, since letters a to f in that column look like hex. Odd digits are another cause: the scan takes complete pairs, so an unpaired digit is dropped and everything after it can shift. Compare the result with Diff bytes against a known good copy.

Your file's name with its extension replaced by .bin. dump.txt becomes dump.bin, and a name with no extension gets .bin appended. The bytes are whatever the hex spelled out, in the order the pairs appeared, with no header, no padding and no trailing newline added. Check the size of the result against what the dump claimed before you trust it.

Yes, if you save it as a text file first. This tool takes a file, not a text box. A useful chain for PEM: run PEM to Hex to get the DER payload as hex text, save that, convert it here, then read the bytes with Parse ASN.1.