The Production Of Knowledge: The Challenge Of Social Science Research

The Production of Knowledge This page intentionally left blank The Production of Knowledge The Challenges of Social Science Research William H. Starbuck 1 3 Great Clarendon Street, Oxford ox2 6dp Oxford University Press is a department of the University of Oxford. It furthers the University’s objective of excellence in research, scholarship, and education by publishing worldwide in Oxford New York Auckland Cape Town Dar es Salaam Hong Kong Karachi Kuala Lumpur Madrid Melbourne Mexico City Nairobi New Delhi Shanghai Taipei Toronto With offices in Argentina Austria Brazil Chile Czech Republic France Greece Guatemala Hungary Italy Japan Poland Portugal Singapore South Korea Switzerland Thailand Turkey Ukraine Vietnam Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries Published in the United States by Oxford University Press Inc., New York ß William H. Starbuck 2006 The moral rights of the authors have been asserted Database right Oxford University Press (maker) First published 2006 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior permission in writing of Oxford University Press, or as expressly permitted by law, or under terms agreed with the appropriate reprographics rights organization. Enquiries concerning reproduction outside the scope of the above should be sent to the Rights Department, Oxford University Press, at the address above You must not circulate this book in any other binding or cover and you must impose the same condition on any acquirer British Library Cataloguing in Publication Data Data available Library of Congress Cataloging in Publication Data Data available Typeset by SPI Publisher Services, Pondicherry, India Printed in Great Britain on acid-free paper by Biddles Ltd., King’s Lynn, Norfolk ISBN 0–19–928853–4 978–0–19–928853–3 1 3 5 7 9 10 8 6 4 2 Contents List of Figures List of Tables vi vii 1. Creating Better Social Science 1 2. A Journey into Disillusionment—Discovering Problems 5 3. Pretences of Research 74 4. A Journey into Hope—Discovering Partial Solutions 95 5. The Production of Knowledge 142 References 171 189 Index v List of Figures 2.1. Mean correlations over time 41 2.2. Percentage improvements with interventions over time 42 2.3. Citing half-lives in years 45 2.4. Percentages of references to the same journal 45 2.5. Numbers of references per article 46 2.6. Correlations reported in three journals 49 2.7. Ratio of forecasted change to actual change (original data) 61 2.8. Ratio of forecasted change to actual change (revised data) 62 2.9. Ratio of forecasted change to actual change for four types of forecasts 63 3.1. Frequency of significance tests in twelve psychological journals 77 4.1. Why diagnosis impedes treatment 109 4.2. Profitability versus market share of US brewers 116 4.3. How observed data misrepresent the present and future 121 4.4. Ratio of fit with OLS regression to fit with equally weighted independent variables 133 4.5. Maximum errors in effect sizes when estimated by OLS regression 134 4.6. Influence of deviations on estimated functions 135 4.7. Ratios of fit with OLS regression and fit with absolute-value regression to fit with equally weighted independent variables 135 5.1. Errors in estimated coefficients 163 5.