The principal component analysis network (PCANet) is an unsupervised deep network, utilizing principal components as convolution filters in its layers. Albeit powerful, the PCANet suffers from two fundamental problems responsible for its performance degradation. First, the principal components transform the data as column vectors (which we call the amalgamated view) and incur a loss of spatial information present in the data. Second, the generalized pooling in the PCANet is unable to incorporate spatial statistics of the natural images, and it also induces redundancy among the features. In this research, we first propose a tensor-factorization-based deep network called the tensor factorization network (TFNet). The TFNet extracts features by preserving the spatial view of the data (which we call the minutiae view). We then proposed HybridNet, which simultaneously extracts information with the two views of the data since their integration can improve the performance of classification systems. Finally, to alleviate the feature redundancy among hybrid features, we propose Attn-HybridNet to perform attention-based feature selection and fusion to improve their discriminability. Classification results on multiple real-world datasets using features extracted by our proposed Attn-HybridNet achieves significantly better performance over other popular baseline methods, demonstrating the effectiveness of the proposed techniques.