You are dealing with two problems here:
1. Image-to-image registration
2. Camera/lens calibration
Something like Hugin may help you to solve both of those problems in one go, but since Hugin is mostly aimed at panorama stitching, you may have some difficulty in getting it to do what you want. Still, that would be my first recommendation.
There are many possible reasons why you cannot align the images with the simple transform tool in PS, but here is a brief explanation of what you are dealing with. Each lens (i.e., 300 mm and telescope) can be characterised by a "camera matrix" (in the lingo), which comprises intrinsic camera (lens) parameters, and extrinsic camera (lens) parameters. Extrinsic camera parameters deal with the rotation and translation of your camera (lens) with respect to a reference target; in your case, you are looking at the set of extrinsic parameters that will move from one coordinate system (say the telescope lens) to the other (the 300 mm lens). If you were careful with the set-up, you can mostly ignore these extrinsic parameters because they will be identical for the two lenses.
The tricky part is dealing with the intrinsic parameters. These include the focal length, principal point (optical centre of lens, which is close, but not quite at, the centre of the image/sensor), and lens distortions. Lens distortions are split into radial and tangential parts; the radial distortions can reproduce many of the well-known photographic distortions such as barrel distortion or pincushion distortion. Tangential distortions deal with whether the sensor is perpendicular to the optical axis of the lens.
What you are doing is to take two lenses with different intrinsic parameters, but near-identical extrinsic parameters (I assume), and trying to apply rotation/translation/scaling to compensate for the differences in intrinsic parameters. Unfortunately, the rotation/translation operations are more closely related to the extrinsic parameters, which is why you are struggling to obtain the desired results. You first have to normalise the intrinsic parameters by undistorting the images from the two different lenses, but to do that you have to obtain the intrinsic parameters first ....
Hugin allows you to compute the camera/lens parameters empirically by analysing overlapping images taken with the same lens/camera. In other words, capture some panorama shots with both the telescope and the 300 mm lens. Then use Hugin to estimate the intrinsic camera (lens) parameters for each lens separately. Next, you should "undistort" the images separately. When that is done for both images, you should be able to overlap them using only rotation/scaling/translation (in PS).
You can "undistort" the images in many ways, but none of the methods I know are trivial to use. LightRoom allows you to apply lens corrections, which should work fine for your 300 mm (which is likely to be in the database), but you will have to construct a lens profile for your telescope by hand, using the intrinsic lens parameters extracted with Hugin. You can even create a custom correction for your own copy of the 300 mm lens, which should be slightly better than the generic one included with LightRoom.
Alternatively, you could perform the "undistort" operation with OpenCV, but this requires some coding. I am sure other tools (ptlens?) exist that will allow you to undistort the images, but it may be difficult to find one that will allow you to provide the intrinsic camera parameters directly.
If you do MATLAB, you can try the CalTech camera calibration toolkit (see for example their explanation of
the camera calibration parameters). This may require some hacking in MATLAB to do what you need here, but it may be easier than the OpenCV route.
If, after reading all of this, you are still determined to get this to work, feel free to ask questions, and I will help where I can.
edit: I just thought of something else. If a lot of time has elapsed between these two images, then you are dealing with a complex "orbit-like" rotation of the stars relative to the camera. If you work quickly, though, and keep the two images close in time, you can fix that as part of the general rotation/translation step, I think. Unless I am wrong in my first assumption that the bulk of the difference between the to images are because the the differences in the intrinsic parameters of the two lenses. But I see from another thread that you already compensate for the motion of the earth with a tracker, so I'd give the Hugin route a try if I were you ....
