A rotation matrix can be represented as
x1 x2 x3
y1 y2 y3
z1 z2 z3
Where (x1, y1, z1) are the first vector (x2, y2, z2), (x3, y3, z3) are the second and third vectors.
Note this matrix is column major oriented.
When you multiply this matrix by an arbitrary vector in space you will get the rotated vector coordinates.
In some situations it is important to know the axis of rotation. Here I am going to present one method to get the rotation axis from rotation matrix.
Consider this matrix which represents a rotation around axis ( 0.7017,0.7017,0 ) , this actually a vector in x-y plane rotated by 45 degree, Like this
The rotation matrix for rotating around this vector by 45 degree is shown below (which I calculated using quaternion)
0.8536 0.146 0.500
0.1460 0.850 -0.500
-0.500 0.500 0.707
When we multiply a arbitrary vector (x,y,z) by this matrix we gets the vector function F as below
(X 0.8536 + y 0.146 + z 0.5) i + ( x 0.146 + y 0.85 – z 0.5 ) j + ( -x 0.5 + y 0.5 + y 0.707 ) k
I,j,k are the base vectors.
Taking curl of this vector function F, we get the axis of rotation.
I calculated the curl of that function and it is (1,-1, 0). After normalizing this vector we can see the result as (.707,-.707, 0),. Earlier we had created the rotation matrix to rotate around the axis ( .707,.707,0). Our answer curl is in the opposite direction of (.707,.707,0) , but it is still correct.
Only problem is with the direction of rotation that is clockwise or anti clock wise..